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CLAIMS 

What is claimed is: 

A computer-implemented method of structural content filtration to reduce the 
number of hypotheses for the location of an active object in a recorded image generated 
by a graphical user interface (GUI) of an application program comprising: 

transforming the recorded image and a corresponding playback image; 

determining a sub-image from the transformed recorded image which corresponds 
to an object to be searched for in the transformed playback image; 

determining a set of points on the transformed playback image which have 
appropriate values for matching the sub-image; and 

filtering hypotheses on the playback image which are more than a selected 
distance from any one of the subset of points. 

2. The method of claim 1, wherein transforming the recorded image and the 
corresponding playback image comprises changing color values of pixels in the images 
into at least one of black and white pixels. 

3. The method of claim 1, wherein the object comprises a non-textual object. 

4. The method of claim 1, wherein the filtered hypotheses are incorrect 
hypotheses for the active object. 

5. The method of claim 1, wherein a hypothesis comprises a contour of an object 
on the playback image which corresponds to a contour of an object on the recorded 
image. 

6. The method of claim 1, wherein filtering hypotheses comprises rejecting a 
hypothesis if the contour of the hypothesis does not have an intersection with any 
rectangle having a top left comer at a point of the subset of points and a predetermined 
width and height. 

7. The method of claim 1, wherein filtering hypotheses comprises rejecting a 
hypothesis if the distance between one of points in the subset of points and a contour of 
the hypothesis is greater than a predetermined distance. 

8. Ail article comprising: a machine accessible medium containing instructions, 
which when executed, result in structural content filtration to reduce the number of 
hypotheses for the location of an active object in a recorded image generated by a 
graphical user interface (GUI) of an application program by 
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transforming the recorded image and a corresponding playback image; 

determining a sub-image from the transformed recorded image which corresponds 
to an object to be searched for in the transformed playback image; 

determining a set of points on the transformed playback image which have 
appropriate values for matching the sub-image; and 

filtering hypotheses on the playback image which are more than a selected 
distance from any one of the subset of points. 

9. The article of claim 8, wherein instructions to transform the recorded image 
and the corresponding playback image comprise instructions to change color values of 
pixels in the images into at least one of black and white pixels. 

10. The article of claim 8, wherein the object comprises a non-textual object. 

11. The article of claim 8, wherein the filtered hypotheses are incorrect 
hypotheses for the active object. 

12. The article of claim 8, wherein a hypothesis comprises a contour of an object 
on the playback image which corresponds to a contour of an object on the recorded 
image. 

13. The article of claim 8, wherein instructions to filter hypotheses comprise 
instructions to reject a hypothesis if the contour of the hypothesis does not have an 
intersection with any rectangle having a top left corner at a point of the subset of points 
and a predetermined width and height. 

14. The article of claim 8, wherein instructions to filter hypotheses comprise 
instructions to reject a hypothesis if the distance between one of points in the subset of 
points and a contour of the hypothesis is greater than a predetermined distance. 

15. A cognitive control framework system for automatically controlling execution 
of an application program having a graphical user interface comprising: 

a recording component adapted to capture user input data and images displayed 
by the graphical user interface during a recording phase of execution of the application 
program, and to analyze the captured user input data and displayed images to generate an 
execution scenario during the recording phase; and 

a playback component adapted to perform image analysis on images displayed by 
the graphical user interface as a result of processing the simulated user input data during 
the playback phase and captured displayed images from the recording phase, the 
playback component being adapted to reduce the number of hypotheses for the location 
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of an active object in a recorded image generated by the graphical user interface by 

transforming the recorded image and a corresponding playback image; 

determining a sub-image from the transformed recorded image which 
corresponds to an object to be searched for in the transformed playback image; 

determining a set of points on the transformed playback image which have 
appropriate values for matching the sub-image; and 

filtering hypotheses on the playback image which are more than a selected 
distance from any one of the subset of points. 

16. The system of claim 15, wherein transforming the recorded image and the 
corresponding playback image comprises changing color values of pixels in the images 
into at least one of black and white pixels. 

17. The system of claim 15, wherein the object comprises a non-textual object. 

18. The system of claim 15, wherein the filtered hypotheses axe incorrect 
hypotheses for the active object. 

19. The system of claim 15, wherein a hypothesis comprises a contour of an 
object on the playback image which corresponds to a contour of an object on the 
recorded image. 

20. The system of claim 15, wherein filtering hypotheses comprises rejecting a 
hypothesis if the contour of the hypothesis does not have an intersection with any 
rectangle having a top left comer at a point of the subset of points and a predetermined 
width and height. 

21. The method of claim 15, wherein filtering hypotheses comprises rejecting a 
hypothesis if the distance between one of points in the subset of points and a contour of 
the hypothesis is greater than a predetermined distance. 



